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Introduction 


This  report  summarizes  the  research  and  development  effort  conducted  under  grant  by  the  National 
Academy  of  Sciences,  the  National  Research  Council  (NRC),  for  the  AFRL  Warfighter  Interface 
Division,  Cognitive  Systems  Branch.  A  full  description  of  the  effort  can  be  found  in  “Behavioral 
Modeling  and  Simulation:  From  Individuals  to  Societies”  (ISBN:  9780309118620)  published  by 
the  National  Academies  Press,  Washington,  DC.  This  Technical  Memo  substitutes  for  the  Technical 
Report  (TR)  to  close  out  this  work  unit  number,  28300413,  in  FIW1S  and  DT1C  since  a  book  was  published 
in  place  of  the  TR. 

Background 

The  Air  Force  and  the  other  military  services  are  increasingly  interested  in  using  models  of  the  behavior 
of  humans,  as  individuals  and  in  groups  of  various  kinds  and  sizes,  to  support  the  development  of 
doctrine,  strategies,  and  tactics  for  dealing  with  state  and  non-state  adversaries,  for  use  in  analysis  of  the 
current  political  and  military  situation,  for  planning  future  operations,  for  training  and  mission  rehearsal, 
and  even  for  the  acquisition  of  new  systems.  This  report  refers  to  this  broad  class  of  models  as  individual, 
organizational,  and  societal  (10S)  models.  There  are  many  lines  of  research  on  such  models,  which  span 
several  disciplines,  have  different  goals,  and  often  use  different  terminologies. 

The  National  Research  Council  was  asked  by  the  U.S.  Air  Force  to  review  relevant  10S  modeling 
research  programs  in  the  various  research  communities,  evaluate  the  strengths  and  weaknesses  of  the 
programs  and  their  methodologies,  determine  which  have  the  greatest  potential  for  military  use,  and 
provide  guidance  for  the  design  of  a  research  program  to  effectively  foster  the  development  of  10S 
models  useful  to  the  military. 

Methodology 

The  formal  statement  of  task  for  the  study  includes  the  following  specific  items: 

•  Review  the  state-of-the-art  of  the  subset  of  the  social  sciences  perceived  as  having  the  greatest 

payoff  in  terms  of  informing  future  computational  model  developments. 

•  Review  the  state-of-the-art  in  societal1  modeling  applications  serving  the  U.S.  Department  of 

Defense  (DoD)  and  related  agencies,  with  special  emphasis  given  to  computational  modeling 
and  simulation  based  approaches. 

•  Review  the  state-of-the-art  in  the  three  computational  modeling  communities  outside  DoD 

(cognitive  science  and  individual  behavioral  modeling,  network  analysis  and  multiagent 
organizational  modeling,  and  multiresolution  modeling  and  simulation)  and  identify  strengths 
and  shortcomings  in  each. 

•  Identify  how  gaps  in  societal  behavioral  modeling  applications  serving  DoD  and  related 

agencies  might  be  filled  by:  conceptual  models  in  the  social  sciences;  computational 
modeling  approaches  now  under  way  in  the  social  science  community;  and  closer  linkages 
between  the  cognitive  science  community,  the  network/organizational  modeling  community, 
and  the  multiresolution  modeling  and  simulation  community. 

1 

In  this  study,  the  committee  broadened  the  scope  to  include  individual  and  organizational  models  as  well,  because  of  the 
inseparability  of  all  three,  given  the  intended  usage. 


1 


•  Develop  a  research-and-development  roadmap  to  fill  current  application  gaps,  for  the  near-, 
mid-,  and  far-term. 

Today’s  military  missions  have  shifted  away  from  force-on-force  warfare — fighting  nation-states  using 
conventional  weapons — toward  combating  insurgents  and  terrorist  networks  in  a  battlespace  in  which  the 
attitudes  and  behaviors  of  civilian  noncombatants  may  be  the  primary  effects  of  military  actions.  These 
new  missions  call  for  agile,  indigenously  sensitive  forces  capable  of  switching  quickly  and  effectively 
from  conventional  combat  to  humanitarian  assistance  and  able  to  defuse  tense  situations  without,  if 
possible,  the  use  of  force.  IOS  models  are  greatly  needed  for  planning,  supporting,  and  training  for  these 
forces  and  for  evaluating  the  technology  with  which  they  fight.  Models  of  human  behavior  in  social 
units — teams,  organizations,  cultural  and  ethnic  groups,  and  societies — are  needed  to  understand,  predict, 
and  influence  the  behavior  of  these  social  units. 

For  example,  models  could  be  used  to  predict  the  effects  of  actions  intended  to  disrupt  terrorist  networks, 
to  predict  the  response  of  insurgents  and  the  local  population  to  the  presence  of  friendly  forces  in  a  given 
area,  or  to  predict  the  effects  of  alternative  diplomatic,  military,  and  economic  courses  of  action  on  the 
attitudes  and  behaviors  of  the  population  in  a  region  of  interest.  Models  could  also  be  used  in  training  and 
mission  rehearsal  to  create  simulation  environments  in  which  military  units  could,  for  example, 
experience  the  effects  of  their  actions  on  the  (simulated)  behavior  of  a  crowd  that  might  either  disperse  or 
turn  hostile.  Models  could  also  be  used  to  evaluate  the  likely  results  of  proposed  changes  intended  to 
make  military  command  and  control  organizational  structures  more  agile  and  adaptive,  and  to  assess  the 
effects  of  introducing  new  technology  capabilities  on  the  performance  of  these  organizations. 

The  NRC  committee  used  a  framework  of  modeling  pitfalls,  lessons  learned,  and  future  needs  to 
characterize  their  major  conclusions  in  a  way  that  will  be  most  useful  to  the  sponsors  in  the  design  of 
future  research  programs.  The  problems  or  pitfalls  identified  by  the  committee  are  organized  in  terms  of 
five  major  categories: 

1.  Modeling  strategy — matching  the  problem  to  the  real  world:  Difficulties  in  this  area  are  created 

either  by  inattention  to  the  real  world  being  modeled  or  by  unrealistic  expectations  about  how 
much  of  the  world  can  be  modeled  and  how  close  a  match  between  model  and  world  is  feasible. 

2.  Verification,  validation,  and  accreditation:  These  important  functions  often  are  made  more 

difficult  by  expectations  that  verification,  validation,  and  accreditation  (VV&A) — as  it  has  been 
defined  for  the  validation  of  models  of  physical  systems — can  be  usefully  applied  to  IOS  models. 

3.  Modeling  tactics — design  ing  the  internal  structure  of  a  model:  Problems  are  sometimes  generated 

by  unwarranted  assumptions  about  the  nature  of  the  social,  organizational,  cultural,  and 
individual  behavior  domains,  and  sometimes  by  a  failure  to  deliberately  and  thoughtfully  match 
the  scope  of  the  model  to  the  scope  of  the  phenomena  to  be  modeled. 

4.  Differences  between  modeling  physical  phenomena  and  human  behavior — dealing  with  uncertainty 

and  adaptation:  Problems  arise  from  unrealistic  expectations  of  how  much  uncertainty  reduction 
is  plausible  in  modeling  human  and  organizational  behavior,  as  well  as  on  poor  choices  in 
handling  the  changing  nature  of  human  structures  and  processes. 

5.  Combining  components  and  federating  models :  Problems  arise  from  the  way  in  which  linkages 

within  and  across  levels  of  analysis  change  the  nature  of  system  operation.  They  occur  when 
creating  multilevel  models  and  when  linking  together  more  specialized  models  of  behavior  into  a 
federation  of  models. 
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To  summarize,  IOS  modeling  is  a  complex,  emerging  science  with  roots  in  many  different  disciplines.  Its 
advancement  requires  that  researchers  maintain  awareness  of  each  other’s  work  and  build  on  each  other’s 
results,  yet  the  multidisciplinary  nature  of  IOS  modeling  has  created  a  fragmented  field.  For  the  field  to 
advance,  researchers  need  better  frameworks  and  forums  in  which  to  compare,  discuss,  and  evaluate  their 
results.  The  field  currently  features  a  multitude  of  complex  models  created  using  different  data  and 
different  theories  to  address  different  problems,  making  comparative  analysis  nearly  impossible.  Common 
datasets  and  challenge  problems  are  needed  in  order  to  learn  which  modeling  approaches  and  sets  of 
variables  are  most  useful  for  specific  types  of  problems. 

Conclusions 

It  seems  clear  there  is  no  single  right  model  and  probably  will  never  be.  The  committee  concluded  that  a 
federated  modeling  approach,  in  which  different  models  at  different  levels  are  linked  together  and 
component  submodels  can  be  swapped  in  and  out,  is  promising  for  attacking  complex  IOS  modeling 
problems.  However,  considerable  research  needs  to  be  done  to  make  this  federated  vision  a  reality. 
Standards,  architectures,  methods,  and  tools  are  needed  to  lower  the  barriers  for  developing,  linking,  and 
validating  federated  models. 

Different  modeling  purposes  require  different  types  of  models.  In  the  committee’s  judgment,  the  purpose 
of  the  model  should  drive  the  appropriate  variables  to  be  included  in  the  model.  To  do  this  successfully 
requires  a  clear  specification  of  model  purpose  and  criteria  for  usefulness  for  that  purpose,  which  in  turn 
requires  that  model  developers  work  closely  with  the  eventual  users  of  the  model. 

The  committee  also  recommended  validation  for  action,  in  which  the  purpose  of  the  model  drives  its 
validation  criteria.  IOS  models  cannot  be  validated  “in  general” — they  must  be  validated  for  a  specific 
use.  A  cross-disciplinary  community  of  interest  needs  to  establish  and  promulgate  accepted  standards  for 
validation  of  IOS  models.  Triangulation  methods  that  combine  expert  judgment,  qualitative  results  and 
theoretical  work,  and  quantitative  results  should  be  further  refined  and  more  widely  used.  Common 
challenge  problems  and  datasets  are  needed  to  facilitate  docking  of  models  for  comparative  purposes. 

Finally,  models  of  human  beings  and  their  individual  and  collective  behaviors  necessarily  include  a  large 
amount  of  inherent  uncertainty.  This  uncertainty  is  not  a  flaw  of  the  model  and  cannot  be  designed  out  of 
the  model.  Human  behavior  is  dynamic  and  adaptive  over  time,  and  it  is  impossible  at  the  moment  (and 
into  the  foreseeable  future)  to  make  reliably  exact  predictions  about  it.  Researchers  need  to  develop  ways 
to  estimate  the  probability  of  plausible  outcomes  and  express  those  estimates  in  ways  that  are  clear  and 
meaningful  to  model  users,  who  can  then  judge  whether  the  results  meet  their  needs.  It  is  important  also 
to  avoid  raising  expectations  about  the  capabilities  of  IOS  models  beyond  what  can  realistically  be 
expected. 

Recommendations 

The  NRC’s  recommendations  for  an  IOS  modeling  research-and-development  program  fall  into  three 
broad  categories:  (1)  large-scale,  integrated  cross-disciplinary  research  programs,  focused  around 
representative  challenge  problems  and  common  datasets;  (2)  research  in  six  independent  areas  that  will 
advance  the  capabilities  to  address  these  integrated  problems;  and  (3)  multidisciplinary  conferences, 
workshops,  and  other  information  exchange  forums,  with  attendees  to  include  not  only  model  developers 
but  also  government  program  managers  and  military  decision  makers. 
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Integrated  Cross-Disciplinary  Research  Programs 

The  committee  suggested  for  the  funding  of  multiple  large-scale,  multiyear  research  programs,  that  focus 
on  comparing  and,  if  appropriate,  integrating  models  from  different  disciplines,  different  perspectives, 
and  different  levels  of  detail  be  considered.  The  goal  would  be  to  create  a  level  playing  field  on  which  the 
capabilities  of  different  approaches  could  be  compared  and  the  strengths  of  each  assessed.  The  ultimate 
goal  is  to  move  10S  modeling  science  forward  through  the  process  of  comparison,  docking,  and 
integration. 

It  is  essential  for  all  participants  in  each  program  to  focus  on  the  same  well  defined  challenge  problem 
instantiated  in  a  common  testbed  and  to  use  a  common  dataset.  At  the  heart  of  each  program  would  be  a 
representative  problem  that  is  critical  for  military  operations,  defined  in  detail.  The  committee  chose  five 
representative  problems  as  a  starting  point  for  choosing  the  problems  to  be  addressed. 

The  research  teams  for  these  efforts  should  be  multidisciplinary,  and  the  program  team  should  also 
include  military  users  with  operational  experience  in  the  domain  for  which  the  models  are  to  be 
developed.  These  users  will  be  ultimate  judges  of  whether  model  results  are  useful  and  will  provide 
advice  on  how  the  results  can  best  be  presented.  The  use  of  a  common  challenge  problem  and  a  common 
testbed  will  facilitate  the  “docking”  of  the  different  models  for  purposes  of  comparison.  The  development 
of  challenge  problems  should  be  a  major  focus  early  in  the  development  of  research  programs. 

These  integrated  programs  will  encourage  mutual  education  between  modelers  and  operational  users. 
Results  should  be  presented  at  workshops  for  program  participants  and  other  interested  parties  and  at 
public  conferences  as  well  as  published  in  the  open  literature. 

Independent  Research  Thrusts 

In  support  of  the  integrated  programs  the  committee  recommended,  they  identified  six  independent  areas 
in  which  research  is  needed.  Progress  in  each  of  these  areas  could  increase  the  ability  to  develop  the 
integrated  modeling  capabilities  that  are  needed  to  address  military  problems.  In  each  area,  it  was 
suggested  to  fund  multiple  research  teams  from  multiple  perspectives,  with  periodic  workshops  for 
researchers  to  exchange  results.  The  committee  also  suggested  that  operational  users  as  well  as 
government  program  managers  participate  in  these  workshops. 

Thrust  1:  Theory  Development 

Models  should  be  conceptually  correct  and  grounded  in  the  underlying  fundamentals  of  what  is  known 
about  individual  human  and  group  social  behavior.  However,  current  theory  in  this  area  does  not  answer 
all  of  the  questions  needed  to  structure  models  that  address  relevant  issues.  Basic  research  is  needed  for 
theory  development,  especially  for  the  low-level  social  behaviors  that  are  the  building  blocks  for  larger 
scale  social  behavioral  patterns.  This  theory  development  work  must  involve  multiple  disciplines  and 
perspectives  with  periodic  workshops  to  exchange  results. 

Theory  development  challenge  problems  should  be  defined  to  guide  the  work,  but  these  can  be 
nonmilitary  and  need  not  involve  the  level  of  military  detail  necessary  for  the  integrated  problems 
discussed  above.  A  series  of  workshops  should  be  conducted  with  researchers  to  identify  key  theory  gaps. 

Academic  institutions  are  key  players  for  theory  development,  but  they  need  information,  incentives,  and 
funding  to  address  these  theoretical  issues.  There  is  a  need  to  educate  researchers  in  military  domains, 
establish  conferences  and  journals,  in  which  their  results  can  be  presented,  provide  postdoctoral  support, 
and  provide  funding  that  allows  researchers  to  spend  time  learning  about  military  domains  in  depth. 
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Thrust  2:  Uncertainty,  Dynamic  Adaptability,  and  Rational  Behavior 

Models  must  deal  with  the  inherent  uncertainty  and  the  dynamic  adaptation  that  characterizes  human 
behavior.  Models  must  also  be  capable  of  modeling  both  rational  and  nonrational  behavior. 

Basic  research  is  needed  in  each  of  these  areas.  Issues  include: 

•  How  should  models  capture  the  “uncertainty-in-the-small”  associated  with  individuals  and 

small  groups?  How  can  model  structures  and  parameters  capture  this  variability,  and  how 
much  of  this  variability  must  be  included  for  the  purposes  of  the  model? 

•  How  should  models  capture  the  “uncertainty-in-the-large”  associated  with  populations  and 

variations  in  population  distributions?  How  much  variability  must  be  included  for  the 
purposes  of  the  model? 

•  How  can  models  capture  adaptation  and  learning  over  time  and  as  the  results  of  actions  by 

others?  For  example,  people  have  multiple  overlapping  identities  and  allegiances.  How 
can  these  be  captured  in  a  model,  and  how  can  one  estimate  the  effects  of  actions  and 
events  on  the  primacy  of  these  multiple  allegiances  as  they  affect  decisions  and  actions? 

•  What  are  the  factors  that  contribute  to  rational,  adaptive  behavior,  and  what  factors  induce 

behavior  that  appears  irrational?  Models  of  both  rational  and  irrational  behavior  must 
capture  all  the  key  factors — cognitive,  affective,  cultural,  and  contextual — that  motivate 
and  shape  behavior  of  specific  individuals  in  specific  situations. 

Better  techniques  are  needed  for  understanding  the  implications  of  diversity  and  variability  for  model- 
based  sensitivity  analysis.  Better  automated  technology  is  needed  to  put  the  model  through  its  paces  to 
explore  the  parameter  space  effectively  and  produce  robust  results. 

Thrust  3:  Data  Collection  Methods 

The  difficulty  of  obtaining  data  is  an  ongoing  challenge  for  10S  modeling.  Research  is  needed  to  develop 
better  data  collection  processes  through  field  studies,  experiments,  and  potentially  massive  multiplayer 
online  games  (MMOGs). 

Although  a  variety  of  ethnographic  data  collection  techniques  are  currently  in  use,  they  need  to  be  better 
tailored  to  the  needs  of  10S  models.  For  field  data  collection,  it  is  necessary  to  bring  modelers  and  data 
collectors  together  to  develop  data  ontologies,  joint  specifications,  and  data  collection  methodologies  and 
tools  that  are  specifically  tuned  to  10S  models. 

MMOGs  are  a  potential  untapped  resource  for  collecting  social  and  behavioral  data  on  a  large  scale.  The 
NRC  panel  recommends  the  creation  of  an  MMOG  facility  and  the  funding  of  basic  research  to  determine 
if  MMOGs  can  be  used  to  test,  verify,  and  validate  10S  models.  The  NRC  recommended  that  funding  be 
put  into  developing  the  science  of  MMOGs.  They  noted  that  funding  MMOGs  is  a  risky  endeavor  but 
think  that  the  potential  benefits  outweigh  the  risks. 

Thrust  4:  Federated  Models 

It  is  a  fundamental  conclusion  of  the  committee  that  no  single  modeling  approach  can  provide  all  the 
capabilities  needed  by  DoD.  They  recommended  a  federated  approach  in  which  modeling  components  are 
created  to  be  interoperable  across  levels  of  aggregation  and  detail.  For  example,  a  federated  model  might 
include  a  detailed  representation  of  a  few  key  individuals,  linked  to  group-level  models  of  different 
cultural  groups  and  terrorist  organizations,  linked  to  geographic  sector-level  models  of  the  level  of  unrest 
in  a  city.  This  approach  is  flexible  and  extensible,  allowing  the  addition  or  subtraction  of  models  at 
different  levels  of  detail  as  needed  for  the  problem  to  be  addressed. 
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Combining  model  components  to  create  federated  models,  in  the  sense  being  recommended,  requires  deep 
semantic  interoperability  (i.e.,  theoretical  consistency)  and  presents  difficult  challenges.  To  create 
semantic  interoperability,  it  is  necessary  to  recognize  that  the  links  among  components  are  themselves 
elements  of  the  model.  Research  is  needed  on: 

•  How  to  ensure  that  the  models  being  federated  embrace  compatible  assumptions  regarding 

concept  abstractions,  entity  resolution,  time-scale  resolution  (tempo),  uncertainty, 
adaptability,  docking  standards,  input/output,  semantics,  etc. 

•  How  the  components  of  the  federated  model  should  be  encapsulated,  and  which  elements 

must  be  exposed  to  other  components. 

•  How  specific  classes  of  models  should  be  linked  (e.g.,  cognitive  models  to  social  network 

models). 

•  How  to  ensure  dynamic  extensibility. 

In  addressing  these  issues,  10S  modelers  should  maintain  awareness  of  research  and  development  in 
model  federation  in  the  larger  modeling  and  simulation  community. 

Thrust  5:  Validation  and  Usefulness 

Current  verification,  validation,  and  accreditation  (VV&A)  concepts  and  practices  were  developed  for  the 
physical  sciences,  and  the  report  argues  that  different  approaches  are  needed  for  10S  models.  Specifically, 
it  recommends  that  a  “validation-for-action”  approach  be  used  that  assesses  the  usefulness  of  a  model  for 
the  specific  purposes  for  which  it  was  developed.  It  is,  thus,  very  important  that  the  purpose(s)  and  criteria 
for  judging  success  be  clearly  stated  a  priority  for  all  models.  The  report  further  recommends  organizing 
national  workshops  to  agree  on  appropriate  processes  for  VV&A  of  10S  models  and  to  outline  a  roadmap 
for  developing  improved  processes  and  standards.  On  the  basis  of  the  results  of  this  workshop,  the 
committee  recommended  that  a  DoD-wide  authority  develop  and  disseminate  VV&A  processes  and 
standards  for  10S  models.  Basing  model  validation  on  the  usefulness  of  the  model  for  specific  problems 
requires  that  model  purposes  be  clearly  stated  by  model  users  and  clearly  understood  by  model 
developers.  They  suggested  that,  as  part  of  developing  a  VV&A  standard  for  10S  models,  clear  guidelines 
be  developed  for  specifying  model  purpose. 

Thrust  6:  Tools  and  Infrastructure  for  Model  Building 

It  is  important  to  reduce  the  barrier  to  entry  for  developing  models,  modeling  tools,  frameworks,  and 
testbeds.  Scientists  should  be  able  to  build  and  validate  models  without  the  large  overhead  currently 
associated  with  many  DoD  modeling  and  simulation  investments.  It  should  be  possible  to  easily  tailor 
existing  models  for  specific  purposes. 

Sharing  of  IOS  modeling  knowledge  across  disciplines,  as  facilitated  by  the  conferences  and  workshops 
recommended  below,  will  support  this  goal.  Work  is  also  needed  in  developing  an  infrastructure  for  IOS 
modelers,  including  a  national  network  of  possible  collaborators,  common  databases  for  model 
development  and  testing,  and  frameworks  and  toolkits  for  rapid  model  development. 

The  limited  data  that  exist  for  IOS  models  are  often  not  accessible  to  model  developers.  The  NRC 
recommends  national  web-accessible  data  repositories  that  are  open  to  researchers  who  seek  to  inform 
and  test  models.  For  militarily-relevant  domains  in  which  some  data  are  classified,  the  committee 
recommends  an  investment  in  automated  tools  to  sanitize  the  data. 
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They  also  recommend  the  development  and  maintenance  of  an  online  web-based  catalog  of  general 
approaches,  models,  simulations,  and  tools.  The  notion  is  to  develop  something  along  the  lines  of 
DMSO’s  Modeling  and  Simulation  Resource  Repository  or  the  clearinghouse  at  Carnegie  Mellon’s 
CASOS  site  (http://www.casos.cs.cmu.edu).  To  be  effective,  the  envisioned  site  needs  careful 
consideration  in  terms  of  organization,  content,  currency,  and  usability.  This  cannot  be  a  one-time  effort. 
It  needs  significant  startup  funding  and  continued  support  over  its  lifetime. 


Multidisciplinary  Conferences  and  Workshops 

A  number  of  the  issues  and  problems  identified  by  the  panel  were  the  results  of  the  failure  of  different 
disciplines  to  exchange  information,  or  they  resulted  from  misunderstandings  among  government  funders 
of  model-development  efforts,  military  users  of  models,  and  model  developers.  Because  of  the  diversity 
of  this  group,  there  is  no  natural  forum  for  them  to  exchange  information.  The  panel  recommended  the 
organization  of  special-purpose  workshops  around  the  integrated  research  programs  recommended  above, 
as  well  as  workshops  for  the  independent  research  thrusts  described  above. 

10S  modelers  need  to  be  educated  on: 

•  The  nature  of  the  military  decisions  for  which  models  are  relevant 

•  Desired  model  functionality 

•  The  most  useful  form  for  presenting  model  results 

•  The  value  of  work  performed  by  others  outside  their  discipline 

•  Feasible  and  appropriate  VV&A  approaches  for  10S  models 
Operational  users  and  managers  need  to  be  educated  on: 

•  The  value  of  multidisciplinary  approaches  and  the  need  for  review  of  models  from  multiple 

perspectives 

•  The  inherent  uncertainty  associated  with  model  predictions 

•  The  value  of  models  for  sensitivity  and  trade-off  analysis  (versus  the  one  right  answer) 

•  The  design  of  virtual  experiments  to  assess  results  over  a  range  of  conditions 

•  Reasonable  definitions  of  validation  for  10S  models,  feasible  approaches  for  VV&A  testing, 

and  why  these  approaches  differ  from  those  used  for  physics-based  models 

The  recommended  workshops  should  involve  model  developers,  operational  military  users  of  the  models, 
and  government  personnel  making  funding  decisions  regarding  model  development 

Roadmap  for  Future  Research  and  Development 

The  committee  recommends  a  use-driven  research  program  to  extend  the  state-of-the-art  in  10S  modeling, 
focused  around  a  series  of  challenge  problems — clear  specifications  of  the  uses  to  which  the  model  is  to 
be  put,  defined  to  be  relevant  to  military  needs,  and  expanded  over  time  as  progress  is  made  in  modeling 
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approaches,  tools,  and  technologies.  The  purpose  of  the  model,  as  captured  in  the  challenge  problems, 
drives  the  theory  to  be  applied,  the  data  to  be  used,  and  the  model  development.  Model  development  is 
made  easier  by  modeling  tools  and  infrastructure  and  relies  on  federation  standards  to  ensure  the 
interoperability  of  model  components.  Once  the  model  is  developed  it  is  validated  by  asking  the  question: 
Is  the  model  useful  for  its  intended  purpose? 

The  recommended  program  proceeds  in  a  cyclical  fashion.  Based  on  the  answers  to  the  question  “Is  the 
model  useful?”  new  models  may  need  to  be  developed,  new  theory  and  new  data  (and  new  types  of  data) 
may  be  needed,  and  new  interoperability  standards,  tools,  and  infrastructure  may  be  required.  Depending 
on  the  results,  the  problem  itself  may  need  to  be  redefined,  clarified,  or  expanded.  These  challenge 
problems,  combined  with  periodic  workshops  and  conferences  to  compare  and  exchange  results,  serve  as 
a  unifying  force  and  a  common  ground  for  the  fragmented  field  of  10S  modeling,  providing  a  foundation 
on  which  scientific  progress  can  be  made. 
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